3.1 Field of the Invention
The exemplary illustrative technology herein relates to systems, software, and methods for making analytical judgments. It is particularly useful for issues that require weighing of alternative explanations of what has happened, is happening, or is likely to happen in the future. The present invention has applications in the areas of business and intelligence analysis, criminal forensics, cognitive psychology, computer science, economics, decision theory, information processing and analysis, and management.
3.2 The Related Art
Analytic activities involve processes to generate hypotheses, to collect and record known relevant information, to categorize relevant information as to diagnosticity, reliability, or other factors, to test hypotheses by comparing the hypothesis against relevant information to determine those hypotheses that are supported by the relevant information and those that are not, and to determine and validate indicators for use in acquiring additional relevant information. Analytic activities can be classified as manual, automation assisted, and automated. Manual analytic activities are those that are performed solely by an analyst, automation assisted analytic activities are performed by an analyst with automation assistance, and automated analytic activities are those activities performed solely by a computer.
Relevant information is information used in analytic activities to determine which hypotheses are likely, and which are not, or to suggest one or more hypotheses to consider. Relevant information can be physical evidence, the information gained from analysis of physical evidence, witness reports, photographs, videos, audio recordings, transcripts of visual or audio recordings, expert testimony, deductions based on other relevant information, computer data, or any other information that can be used to support one or more hypotheses, to show lack of support for one or more hypotheses, or to suggest one or more possible hypothesis. Where relevant information is absent, but might be expected to be present, the lack of relevant information can also constitute relevant information. For example, if an aircraft was stolen from an airfield, it would be expected that the tower records would show a departure by that aircraft around the time it went missing from the airfield. If there is no such departure located, that lack is relevant information in itself, and might support hypotheses that the aircraft was hidden at the airfield rather than stolen, that it was disassembled and removed by truck, or that it was never present in the first place, while at the same time reducing support for hypotheses that include the idea of the aircraft being flown away by thieves.
Indicators are observable, or potentially observable, actions, conditions, or events that can be monitored to collect relevant information over time. Specific indicators occurring or reaching pre-determined values will support a conclusion that one or more specific hypotheses has happened, is happening, or is becoming more likely to happen, while if they do not occur or do not reach the pre-determined values, will support a conclusion that one or more hypotheses did not happen, are not happening, or are less likely to happen.
Analytic activities typically start by generating a set of hypotheses. The set of hypotheses generated ideally includes all reasonable hypotheses. There are a number of known manual techniques for generating hypotheses, including, but not limited to, Structured Brainstorming, Nominal Group Technique, the Delphi Method, Multiple Hypotheses Generation and Quadrant Hypothesis Generation. The specific manual hypothesis generation technique selected varies with the training of the analyst(s), and to some degree, the appropriateness of the technique to the situation. There are no known examples of automated or automation-assisted hypothesis generation. Given the number of steps and calculations involved in carrying out some of the manual techniques, and the amount of information involved in some steps, the chance for analysts to make errors is high. Automation of hypothesis generation would help to reduce the chance for such errors as well as easing analyst workload and significantly speeding up the analysis processes.
When generating hypotheses, it is necessary to avoid various types of bias that analysts are prone to which can limit or distort the scope of the generated hypotheses and adversely impact the conclusions reached. Some techniques for hypothesis generation have been developed help to avoid some types of bias, but introduce other types of bias. Ways to avoid or limit the effects of bias are needed.
When testing hypotheses as part of analytic activities, it is also necessary to avoid various types of bias. Structured analytic techniques, such as Analysis of Competing Hypotheses (ACH), have been developed to reduce some bias effects. Using the ACH technique manually is tedious and repetitive, time consuming, does not scale well for large numbers of hypotheses and relevant information due to the increasing size of the matrix that results, and does not deal well with a plurality of analysts since they must either share one matrix and agree on consistency ratings, or work individually and then manually merge their consistency ratings or debate their individual conclusions afterward. When analysts are co-located, the need to share a single matrix, or manually merge separate results, can also result in “groupthink” bias, as some analysts are improperly influenced in their determinations by the opinions of other analysts for reasons such as seniority, respect, dislike or other factors.
Each of these techniques can be complex and are slow and awkward to implement manually without error due to the quantity of information involved and the number of steps and calculations needed. Existing automation-assisted ACH programs, such as Open Source ACH, address the mechanics of the data recording aspects of the technique, and perform some of the calculations required. These programs accentuate biases, such as “anchoring” (i.e. fixating on a first reasonable choice and comparing subsequent choices to it). They do not support ways to reduce bias effects such as anchoring or “groupthink”, do not support the compartmentalization of information, nor do they support automated mechanisms for generating hypotheses, do not permit flexible weighting of inputs by analysts (for example, to allow for varying levels of experience of the analyst), nor support distinguishing analysts and results reflecting domain-specific knowledge, and do not support other aspects of analytic activities, such as identification and evaluation of indicators, or generation of hypotheses, nor do they provide means to track analyst progress in rating consistency of hypotheses with relevant information, especially when analysts work independently in separate matrices. When performing manual analytic activities, collaboration between analysts typically requires that they be co-located, both for communication and to have access to the working materials, such as white boards, charts, papers, and other means used to record and organize information.
When collaborating during manual analytic activities, it can be difficult or impossible to maintain compartmentalization of information. Systems and devices to enable easier collaboration between analysts, whether co-located or in diverse locations, while maintaining proper compartmentalization of information, are needed.
Hypotheses or indicators that are common to more than one analytic technique must be manually copied or entered each time a different analytic technique is used to work with them. Doing so with pencil and paper, or even computerized spreadsheets, is awkward, time consuming and prone to error and does not support shared collaborations and compartmentalization of information. Analyst notes, assumptions, or discussions are not retained or associated with specific information, or even recorded in the first place, making it difficult or impossible to obtain a complete view of the history of a hypothesis, indicator, or item of relevant information. Such historical views of these items can provide insight useful for evaluating the quality of the ultimate conclusions of an analytic project. A means of automating and integrating the analytic techniques, with automation to reduce the workload required to implement the individual techniques, that collects and retains historical information about the origin and handling of important aspects of the analysis, is needed to improve the usability of the analysis processes, as well as to increase the quality of results.
Extensible, automated systems are needed for hypothesis generation, hypothesis recording, relevant information recording, hypothesis and relevant information sharing, hypothesis evaluation, indicator recording and evaluation, and analytic history recording, all while maintaining required compartmentalization of information. Automated and automation-assisted methods are needed to reduce analyst workload, reduce the likelihood of errors, to assist with identification and recognition of important relationships, such as which hypotheses a given piece of relevant information relates to, which hypotheses are inconsistent with what relevant information, or the reliability of a given piece of relevant information or its source.
The present invention addresses these and other needs.